Papers with adaptive gradient gating(AGG
Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have determined that the learned token embeddings of large-scale neural language models are degenerated to be anisotropic with a narrow-cone shape. |
| Approach: | They propose a method to degenerate the learning gradient for rare token embeddings by gating the specific part of the gradient for all tokens during training stage. |
| Outcome: | The proposed method improves the performance of the models but lacks the training dynamics needed to solve the representation degeneration problem. |